Automated Testing of Embedded Systems

By Laura Dominik

---

How can manufacturers of embedded systems stay abreast of the growing, increasingly complex, software-intensive technology? How can they continue to produce their products with a shorter time-to-market window and still maintain highly reliable, quality products? The answer--more thorough testing. But more thorough testing, using traditional methods, means an increase in the time allocated to product development. Is there a solution?

The Good News
There is a solution. Test automation can assist manufacturers in obtaining more thorough test coverage through rigorous, controlled, and repeatable tests. These comprehensive tests can then provide faster, more complete product verification within a short time-to-market window. Designers can achieve more thorough test coverage by automating the test and verification process. Automation can assist in producing a set of comprehensive tests that can be performed consistently. In many instances, an automated closed-loop environment can perform testing that manual testing couldnít accomplish or would take too long to perform.

To receive all advantages and benefits provided by an automated test tool, it is important to plan automation into the test and verification process early in the product development cycle.

The purpose of automated testing is to increase test coverage, thus increasing the quality and reliability of the embedded system. More thorough test coverage is achieved through more extensive and better testing. There are many tools available for software testing, but not all of them can provide a solution for embedded software, given its real-time and deterministic aspects. Specifically, an automated test system for embedded devices should provide a platform for real-time testing of the software and its behavior within the product. In order to do that, the automated test system needs to be nonintrusive and capable of real-time stimulation and monitoring of the embedded deviceís internal data flows. Real-time testing ensures that verification is realistic and usage-based. The nonintrusive, real-time monitoring and capture of the target systemís internal data flows translates into testing that does not alter the behavior of the product and thus invalidate the test results. If the test tool is intrusive, it imposes an artificial environment and the behavior of such functions as interrupts, multitasking, and cache memories may be compromised. Nonintrusiveness means no impact on system performance, no timing delays imposed by the test system, and no additional code residing in the target systemís memory. The internal data flows can include microprocessor bus cycles, memory reads and writes, accesses to and from I/O, graphical images, and communications ports. The Benefits of Test Automation There are important and significant differences between manual and automated testing. Manual testing of embedded systems is most useful in situations where the results of a specific and limited set of test cases are needed relatively quickly. Automated testing, on the other hand, requires a greater initial effort to plan, organize, and produce the tests. The key to achieving a high return on investment in test automation and automated test tools is to plan very carefully. One of the reasons manual testing continues to be performed is because of the relatively quick feedback it produces. Automation requires an initial investment of time. Advantages of automation are reduced test cycle calendar time and reduced staff time to run the tests. A reduction in test run time increases test development time. Increased test development time results in increased test coverage quality and earlier defect detection and correction. It also affords the ability to conduct more extensive testing by running a greater number of test cases, thus increasing the quality and reliability of the products. Other advantages of automation include repeatability, leveraging and accumulation. Automated tests can be run repeatedly and consistently, leading to time savings as well as predictability. The benefits are even greater considering the fact that testing is run multiple times for debugging the tests themselves, testing the product, retesting if a bug is found, and retesting again to check the bug fixes. Leveraging automated tests comes not only from repeating a test that was performed manually, but also from executing tests that were never performed manually at all. Accumulation of tests is the most critical benefit for the long term. Applications change and gain complexity over their useful lives. Therefore, the number of tests needed for coverage is constantly increasing, requiring maintainable and reusable tests. And once the tests are developed, long-term cost benefits are derived by editing and reusing them in future projects.

The Problem
First consider the intricacies of embedded electronics. Characteristics that are commonly found in embedded electronics include: stand-alone computer(s)--made up of one or more microprocessors, controllers, and DSPs--running proprietary or standard real-time operating systems. In addition to these common characteristics, advanced functionality is being incorporated into embedded devices, such as network interfaces and graphical operator interfaces. The applications themselves are often real-time, interrupt-driven, multitasking, and I/O-intensive. It is also typical for an embedded application to monitor and control multiple physical variables as they occur. These event-driven real-time applications are characterized by essentially unpredictable combinations of events requiring handling by a computer system.

Also, a large portion of a productís capabilities lie within its software. Todayís software is often designed and developed in modules. An application is composed of many interrelated components, meaning changes in one module can often affect one or more other modules. These increasingly sophisticated software applications bring additional testing and verification challenges. Software defects can occur at various points in the development cycle: incorrect or incomplete requirements, source code errors, or performance faults. Sophisticated testing and verification of the requirements, code, and integrated products is a must in order to find and fix these defects at the earliest possible stage of the development cycle.

Manufacturers of these software-based systems all face the same challenge ó in order to be productive and competitive they must produce a reliable, high-quality product within a short time-to-market window. Consumers and end-users expect and will settle for nothing less. This requirement translates into a need for finding and fixing errors earlier in the development cycle as well as more thorough test coverage.

Embedded systems have response time constraints which add to the complexity of the test and verification of these products and must be considered when selecting a test method and tool. To effectively and efficiently test embedded software, the testing needs to be repeatable, thorough, and traced back to product requirements. To ensure that all product requirements are met, thorough and complete documentation of the tests and results is necessary. By automating the testing of their embedded devices, manufacturers can create more complete, rigorous and repeatable tests for more thorough test coverage. Such tests help produce high-quality products while meeting the time-to-market window.

Solutions for Automated Testing of Embedded Systems
Figure 1 depicts an automated closed-loop testing environment configured to stimulate inputs, checks outputs and log results. Closed-loop testing is important in most critical applications because of the need to verify the target systemís reaction and timeliness to various combinations of input conditions.

Figure 1: Automated closed-loop test environment

In this closed-loop test system, a set of input conditions stimulate the target system into some action. An independent monitoring function then verifies that the target system has responded correctly to the set of input conditions. Note that because of the response-time requirements, a failure for an embedded system is determined not only by the output being incorrect, but also by response time not meeting specification.

The stimulation and monitoring functions are tightly coupled and, through scripting, provide input to cause a target action, monitor the resulting reaction and produce another action. This additional action is based either on the previous reaction or the next test requirement.

The test script sets up various input states to the target, gathers monitored data and performs an analysis of the data. In addition, the script controls the visualization of the device event, the selected simulations and stimulations of device components and inputs, and the automated logging and documentation of test results. The programmability of scripted testing provides push-button control of test execution, and facilitates logging and status reporting. It is also a modular and reusable approach to testing.

The following is an example of how a medical device manufacturer utilized this type of test tool to automate the test and verification of their system.

Industry Solution: Medical Device
The development challenge for medical device electronics involves many issues, including ever increasing complexity and life-critical functionality. Medical device manufacturers are faced with such dilemmas as their product being too complicated to test manually, the high cost of manual testing, and requirements for documented proof of verification. These are some of the very practical reasons that automation of software testing is being utilized.

The following describes the challenges faced by the manufacturer of a dialysis machine which performed continuous realtime analysis of a patientís condition. The dialysis device was a realtime X86-based system with several subsystems, serial communications, a touchscreen, and a keypad. The subsystems read, wrote and processed discrete, analog and keypad I/O. Each system was designed to utilize multiple configurations of the embedded software.

The particular challenge faced by this manufacturer was the excessive amount of time required to test all possible configurations of software using the same hardware platform. The objective of automating the test process was to provide more thorough, efficient testing of the multiple configurations of software. To achieve the desired test objective, a test environment was needed which could automatically stimulate these external signals, monitor output and system data, and document test results.

For monitoring the internal data flows of the dialysis device, the medical device manufacturer chose the B-Tree system (see Figure 2). It consists of a host computer, a microprocessor probe, a stimulation engine, and communications and graphics capture subsystems. The probe, which has a direct connection to the microprocessor pins, collected system data including memory and I/O reads and writes, code fetch activity, and interrupt acknowledge cycles. Graphics display data was captured and compared with the expected display bit maps at the pixel level. Serial communications, keypad operations, touchscreen activity and LED signals were stimulated and monitored. All of the capture, monitoring, and stimulation functions were performed in real time.

Figure 2: Dialysis machine with interface to B-Tree system

The test design phase of the process identified the types of test to run, including functional, safety, error, and boundary testing. Test suites--sets of tests related either by function or the area of the application they impact--were developed next.

The actual testing was performed through scripts written in C. The programmability of scripting provided a modular and reusable approach to test case development. For instance, the for loops allowed for multiple values to be tested and results recorded without additional scripting.

The scripting effort began with the development of libraries of reusable functions. Once the libraries were completed, the scripts were written. The scripts used the libraries to perform the actual steps of the tests. The scripts provided test execution, automatic logging, and status reporting. In particular, they controlled the visualization of device events, the selected simulations and stimulations of device components and inputs, and the documentation of test results. It was at post-run time that the results were examined, automatically compared with the expected values, and were determined to be a pass or fail.

The actual amount of test cases that were performed was significantly increased. And with the simulation capabilities of the test system, tests were run which had previously been difficult or impossible to perform manually. The specific types of tests performed included many combinations of input and input timings, which exercised the system in as many ways as possible. This was to ensure acceptable system behavior for both expected and ìunexpectedî inputs.

Comprehensive verification was achieved by testing the behavior of the software within the system through the simulation of its external environment and monitoring responses. By extensively testing the system inputs, outputs, and their timeliness and interactions, the reliability of the system was validated. This included usage based testing as well as boundary, error, and hazard testing.

The manufacturerís met their objective of more thorough and efficient testing. The tedious and time-consuming manual testing was replaced by automated functional system testing, executed through re-usable, modular test scripts. The ìblack boxî testing verified correct system response to real-time stimulus. And finally, the same scripted tests were used to perform verification and validation on multiple software configurations. Reuse of the test scripts resulted in large time and cost savings throughout the life of the product.